[core] Use full-range anchor as row representative in DE read-type pruning - #9899
Conversation
…uning When every file in a data-evolution row-id-range group lacks the columns of the query projection (e.g. a freshly added column), pruneByReadType keeps one file as a row-count representative so the reader can emit the right number of NULL-filled rows. The representative was group.get(0), which can be a blob or vector-store file: such files cover only a sub-range of the group's row ids, so the split silently emitted fewer rows than the group contains. Use retrieveAnchorFile instead, which selects the oldest full-range normal file and never a blob/vector-store file. The pruning core is extracted into a static method (mirroring evolutionStats) so the representative choice can be pinned by a unit test with a blob file ordered first in the group. Assisted-by: GLM-5.3
|
Right fix. The old fallback Small nit: |
The row-count-representative fallback now calls retrieveAnchorFile with deletion vectors disabled, so the message should not read as DV-only.
|
Done in e1197d2: reworded the |
|
Thank you @JingsongLi |
2 participants
Purpose
close #9898
When a data-evolution row-id-range group has no file that writes the query's projected columns, read-type pruning keeps one file as a row-count representative so the reader emits the right number of NULL-filled rows. The representative was
group.get(0), which can be a blob or vector-store file. Those files cover only a sub-range of the group's row ids, so the split silently emitted fewer rows than the group actually contains. This usesretrieveAnchorFileinstead, which returns a full-range normal data file and never a blob or vector-store file (the same primitive the deletion-vector anchor path already relies on).The pruning core is extracted into a
@VisibleForTestingstatic method so the representative choice can be pinned by a unit test with a blob file ordered first in the group. The filter-field logic added by #9872 is preserved: the instance method still folds in the fields referenced only by the filter and passes them to the static method, so the set of kept files is unchanged.Tests
DataEvolutionFileStoreScanTest#testReadTypePruningKeepsAnchorAsRowRepresentativebuilds a group of a blob file covering rows 0-1 (ordered first) plus two full-range normal files covering rows 0-9, with areadTypereferencing a column no file writes. It asserts the representative is the full-range normal file. Against the oldgroup.get(0)fallback the representative is the blob file, so the assertion fails.